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ABSTRACT 


The author evaluates LeatherNet, a Distributed Interactive Simulation 
compliant, virtual simulation system being developed by the Advanced Research Projects 
Agency to demonstrate Modeling and Simulation(M&S) technologies and to partially fulfill 
the U. S. Marine Corps M&S goals. The research focuses on evaluation of LeatherNet as a 
mission planning and briefing tool for Marine infantry company commanders, staff, and 
subordinate leaders. Evaluation is based on user perception and user performance on a live 
fire range subsequent to using the system. The user surveys indicate high user acceptance 
and belief that LeatherNet is a valuable mission planning and briefing tool and that 
LeatherNet has a good potential to be an effective training tool for commanders and their 
staffs. User performance, evaluated by subject matter experts on a live fire range, showed 
no statistically significant improvement for groups exposed to LeatherNet when compared 
to groups that did not use LeatherNet. The author eplains why true differences, even if they 
do exist, would be difficult to detect due to the lack of experimental control and 
recommends action to be taken by the Marine Corps to conduct further testing with greater 
experimental control. The author also suggests steps the Marine Corps can take to optimize 


its investment in M&S. 
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I. INTRODUCTION 


A. RESEARCH OBJECTIVE 

The purpose of this thesis is to determine the training effectiveness of using 
LeatherNet, a virtual simulation system that models several Marine Corps live fire and 
maneuver ranges, to train infantry company command element staff and subordinate leaders. 
The evaluation specifically addresses LeatherNet as a tool to conduct mission briefing and 
mission planning. Training effectiveness is measured through user performance evaluations, 
conducted by subject matter experts on live fire and maneuver exercises, after user exposure 
to the LeatherNet system. User perception is also used to determine training effectiveness 


as measured through user surveys. 


B. SCOPE OF THIS STUDY 

The scope of this evaluation is limited to testing the training effectiveness of using 
LeatherNet as a mission planning and mission briefing tool. To make a comprehensive 
assessment of the training effectiveness of LeatherNet, with a high degree of accuracy, 
would require many factors not available at the time of this study. First, the system is not 
mature enough to allow comprehensive testing of some system capabilities such as 
interactive force on force war gaming for mission rehearsal. Second, many features of 
LeatherNet that might have been tested, were not, due to user resistance to features they felt 
did not behave realistically or were too time consuming to use. Thirdly, funding and 
personnel to conduct a true experimental test could not reasonably be made available, nor 
is such a test desirable for this particular stage of development of LeatherNet. Determining 
the training effectiveness of military simulation systems is difficult, complex, and costly, 
particularly for systems designed for collective training. Therefore, the expense of 
conducting a true experiment on a demonstration project still under development cannot be 
justified. The user survey and user performance evaluation are intended only as indicators 


of user acceptance and as a preliminary evaluation of the system's value for future reference. 





Cc. MILITARY SIMULATION DEVELOPMENTS | 

Short of combat, all training, mission rehearsals, and exercises conducted in 
preparation for war can be considered simulation. With the advance of computer 
technologies, many of the functions of field training can now be further simulated using 
virtual simulation rather than using live participants in field exercises to simulate the actions 
of combatants. The U.S. military is more reliant upon simulation than is generally realized. 
Development and testing of new weapons such as aircraft, tanks, and ships are accomplished 
with extensive use of simulation to test their value. Furthermore, the crews of many weapons 
systems now rely heavily on simulators for training, and commanders can develop and test 
battle plans and tactics on simulators using combat models. Simulation has the potential to 
be used for a wide range of applications including training, testing, evaluation, mission 
rehearsal, and system acquisition [Ref. 1: p. 1-2]. 

The U.S. military employs three forms of simulation to train its forces and portray 
what happens when one military force engages another: (1) constructive simulation which 
uses mathematical models of combat to simulate warfare for everything from a single 
weapons system to national warfare, (2) live simulation involving real military forces and 
simulated weapons effects, and (3) virtual simulation using computers to simulate warfare 
in synthetic, computer generated environments. [Ref. 1: p. I-4] 

i; National Defense Issues 

Since the end of the Cold War, several trends have provided impetus to increase 
reliance upon modeling and simulation within the Department of Defense (DoD). Although 
maintaining combat readiness is essential for the military, resources devoted to maintain 
combat readiness have steadily declined since the demise of the Soviet threat. The decline 
can be expected to continue for the foreseeable future as the nation continues to grapple with 
the federal budget deficit. Second, despite the diminishing resources, the DoD has seen no 
lessening of its operational tempo, and in fact, has experienced a proliferation of missions 
and contingencies. As funding has diminished, future readiness in terms of recapitalization 
and modernization is at risk of being sacrificed to pay for near term commitments. [Ref. 2: 


p. 2-6] 





While the threat of a major East/West confrontation has subsided, smaller scale 
regional contingencies have multiplied as nationalistic movements, ethnic disputes, and 
rivalries once suppressed by communist regimes continue to inflame the former Warsaw 
Pact countries. Additional challenges are posed by countries such as Iraq and North Korea 
which threaten the security of their neighboring countries, while countries such as Libya, 
China and Iran pose less imminent threats. In addition to containing the spread of regional 
disputes, the U.S. military must also be prepared to combat terrorism and the use of nuclear, 
biological, or nuclear agents. Ongoing efforts to obtain these agents from the stockpiles of 
the old Warsaw pact countries, or to manufacture them in various hostile Third World 
nations present a significant and growing threat to the U.S. and its allies. The combination 
of these less predictable threats in far flung areas of the globe increase the difficulty of 
maintaining a military capability to deal with them. [Ref. 3] 

As part of the current national military strategy, the military has increasingly been 
used for peacetime engagements or Operations Other Than War (OOTW). In order to 
promote regional security and stability, U.S. forces have frequently been called upon to 
conduct nation building, security assistance, humanitarian, counter drug/terrorism, and 
peacekeeping missions. Recent operations in Haiti, Rwanda, and Somalia, as well as the 
current operations in Bosnia demonstrate such OOTW. [Ref. 2: p. 6-12] 

Not only have resources diminished, encroachment by civilian communities upon 
military bases, continued base closing in the U.S., and the loss of overseas bases and training 
areas as the U.S. reduces its overseas military presence, have further constrained the 
military's training opportunities. The loss of training areas is further aggravated by increased 
environmental regulations and endangered species protection which erode the military's 
unlimited access and use of many of its remaining traming areas. 

The increasing cost of training is another constraint that limits the ability of the DoD 
to maintain high readiness levels. As weapons systems have become more sophisticated with 
the use of precision munitions and high technology weapons platforms, both the cost of the 
munitions and the weapons systems have increased dramatically. It is not uncommon for 


Soldiers and Marines to be allowed a single, or even no, live fire shots in a year's time from 





some of the more sophisticated missile systems for which they are crew members. Even 
some of the more common anti-tank missile systems such as the TOW and Dragon systems 
have munitions costs that typically limit the expenditure of live missiles for training 
purposes to one or two missiles per year. Many weapons platforms, such as the M1 Abrams 
tank and the F-18 aircraft, are increasingly expensive to operate, due to fuel and maintenance 
costs, regardless of the amount of training munitions expended. The net result of these cost 
increases in an era of reduced funding is that the services are often forced to balance 
between readiness training and investment in force structure. [Ref. 4: p. 2-4] 

2. Simulation Benefits 

As the Commandant of the Marine Corps pointed out in his planning guidance, in 
an era of diminishing resources, history has shown that Marines can survive both as 
individuals, and as an institution, if they remain highly trained and ready. He notes that the 
Marine Corps has always turned to training and education systems to keep its warfighting 
edge. At several points in his guidance he makes reference to his general statement that "the 
use of simulation, virtual reality models, and various warfighting games can make 
subsequent field training more effective." [Ref. 5] 

In short, modeling and simulation can be for the U.S. Armed Forces what gliders and 
cardboard mockup tanks were to the German Wehrmacht prior to World War I, when more 
attractive and realistic training options were prohibited. Despite constraints being placed on 
the military's ability to conduct training in the traditional manner, large scale use of virtual 
simulation, imagination, and resourcefulness in implementing the emerging modeling and 
simulation technologies, can help enable the DoD to avoid unpreparedness by augmenting 
live training and thus providing the basis of future battlefield success. 

In the context of current and expected future developments within the DoD and the 
U. S. Marine Corps, several assumptions have been made by the Marine Corps Modeling and 
Simulation Management Office (MCMSMO) that drive the need for increased reliance on 
modeling and simulation. The most significant of these are as follows: 

1. That resources for DoD will be constrained for the foreseeable future. 


2. That DoD will continue to develop joint tactics, exercises and operations to 
include joint simulation. 





3. That the cost of weapons platforms and munitions will continue to increase as they 

become more sophisticated. 

4. That training areas will continue to have constraints imposed upon their use by 

civilian communities and agencies [Ref. 6: p. 1-6] 

Despite diminishing resources, the military's investment in simulation will continue 
to expand. Within the Marine Corps, modeling and simulation technologies will be used for 
many varied applications to include validation of requirements and doctrine, testing and 
evaluation, training combat forces, improvement of acquisition processes, and mission 
planning, preview and rehearsal. [Ref. 7] 

The primary reason for the use of simulation is economic. Virtual simulation for 
example, allows commanders and staffs to experiment without the cost of fuel, ammunition, 
equipment maintenance, and large numbers of personnel. Training can be accomplished at 
a small fraction of the cost of using real systems in the field. A single TOW anti-tank 
missile costs $11,500, but a single TOW shot in the Precision Gunnery Training System 
(PGTS) costs only $.05 [Ref. 21]. Likewise, the cost of operating an M1 Abrams tank is 
approximately $92.00 per mile and transit to a gunnery range for a tank platoon can cost 
thousands of dollars, but the cost of the Unit Conduct of Fire Trainer (UCOFT) is only $.25 
per hour. [Ref. 8: p. 18] 

Many simulators such as the UCOFT provide training enhancement in addition to 
the cost savings. Various studies of the UCOFT have found that tank crews trained on the 
system experience substantial gains in the percent of targets acquired, engaged, hit, and 
destroyed and that they open fire with their first round 25% faster than conventionally 
trained gunners [Ref. 21]. The Indoor Simulated Marksmanship Trainer (ISMT) can 
similarly improve training that can be difficult if not impossible to replicate on a real range. 
For the M203 grenade launcher, for example, the system provides detailed, accurate 
measurements of the gunners sight picture and sight alignment for each shot. Thus, 
deficiencies that would normally go unnoticed can be coached. [Ref. 8: p. 19] 

Linking of various simulators allows training to be distributed over great distance 
at minimal cost. Since networked simulators share common databases, participants using 


the simulators can "see" the same visual images from their relative positions in the terrain 





database. Entities, that represent individuals, units, vehicles, or aircraft can see the same 
targets, and each other regardless of the location of the simulator, and more importantly, the 
personnel participating in the simulation. The result is significant cost saving by avoiding 
the need to physically bring forces together to train together. 

As an indicator of the savings possible, in the 1992 REFORGER (Return of Forces 
to Germany) exercise, the Army used constructive simulation to avoid 34 million dollars in 
costs as compared the same exercise in 1988 without simulation [Ref. 9]. The 1992 exercise 
allowed more commands to participate in the event, although the number of troops 
participating in the live exercise was reduced. The net result was that many units had the 
benefit of participating in an exercise without the expense of transporting all the personnel 
and equipment typically necessary. 

Virtual simulators provide the benefit of allowing the user to experience the virtual 
model of the real world he would not otherwise be able to experience due to fiscal 
constraints, occupation by hostile forces, or simply being too dangerous for peacetime 
operations [Ref. 9]. Using the virtual model, the user can freely navigate the terrain, viewing 
the environment from both the friendly and enemy perspectives, conduct mission planning, 
and conduct war gaming against a synthetic enemy, to mention just a few of the options 
available to the user. 

With the growth of environmental concerns upon our existing training areas, not to 
mention the loss of training areas, modeling and simulation technologies offer a viable 
alternative while ensuring the protection of both endangered species and encroaching 
civilian communities. Constructive and virtual simulation require little, if any, deployment 
of troops to the field, and they can act as a mechanism to reduce the size of the live 
simulation force without sacrificing the training objectives. Virtual and constructive 
simulations leave behind no dud ordnance, fuel spills, or environmental degradation like that 
typically resulting from large scale field exercises. 

3. Simulation Limitations 

The benefits of virtual simulation are many but several drawbacks or potential 


drawbacks do exist. The possibility exists that as the use of simulation increases, reduction 
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in training on the actual equipment and reduced participation in live simulations can also 
reduce both the user's confidence in the use of the actual equipment and the military 
readiness of the force. Additionally, if the behaviors of a simulator do not accurately model 
the real world, the simulator may instill habits that are incorrect for use with real scenarios 
and equipment. Accurate models of the characteristics of weapons systems, individual 
combatants, command and control structures, various sensors, and weather, to mention a 
few of the elements requiring modeling in military simulators, are difficult to attain. 
Furthermore, the modeling data may not exist or may be prohibitively expensive to develop. 
The use of funds for development and procurement of simulation systems of unknown value 
can be a dubious undertaking that has the potential to detract from military readiness. In 
addition, the value of a simulation system cannot easily be determined until the system 
reaches mature stages of its development and user testing takes place, which is after the bulk 
of the development costs have been incurred. Finally, the potential exists for the compromise 
of security if access is gained to networks that support interactive simulation, exposing 
sensitive tactics and weapons capabilities. [Ref. 1: p. 1-14] 

4. Simulation Technologies 

Several factors are driving the need for ever greater reliance upon virtual simulation. 
As previously mentioned, the ability to conduct live simulation will be fiscally constrained 
in an era of military downsizing, exerting pressure to reduce the size and number of live 
simulations. Reduction in the size and number of available training areas, and the inability 
to adequately simulate weapons effects on live personnel and equipment are also prime 
factors driving the need for virtual simulation. 

Perhaps the most significant driver increasing the role of simulation is the growth 
of simulation capabilities and related technologies. These new technologies are rapidly 
changing the manner in which the U.S. military trains, deploys, and executes its missions. 
Precision munitions, satellite-based surveillance and communication systems, and 
computer-based mission planning systems are but a few examples that are dramatically 
increasing the information processing capabilities and have served to expedite decision 


making and execution on the battlefield. The rapid growth in computer capabilities, 





including advances in computer-based graphics and communication technologies, have 
enabled the use of networks of interactive simulators to distribute training over great 
distances [Ref. 10: p. 14-15]. Today's simulators can realistically model military entities, 
environments, and combat against synthetic opposing forces. Once individuals and units 
have been trained on the real weapons systems and tactics, further training can be conducted 
on such current systems as the Army's SIMNET (Simulator Networking) using the emerging 
communication systems such as the Distributed Interactive Simulation (DIS) network to link 
simulators. Such networked simulators are designed for unit and joint training in large scale, 
two-sided engagements with various weapons types and command and control groups. 
Networked, interactive simulation systems currently being developed and fielded include 
both British and German versions of SIMNET, the Close Combat Tactical Trainer (CCTT), 
and the Synthetic Theater of War (STOW) demonstration project. [Ref. 1: Chap. I, p. 10] 

a. Distributed Interactive Simulation 

DIS is a follow-on development of SIMNET technology. Once fully 
developed, DIS will support real time communications between networks of various types 
of simulators, in multiple locations, to create a realistic synthetic environment to perform 
interactive training, design and prototyping, war gaming, and other applications. The 
infrastructure will be designed to allow interactive sessions between simulation systems from 
all the services, built for separate purposes, with different technologies, and from different 
vendors. DIS exercises will incorporate a mixture of live, virtual, and constructive entities 
and allow units of all sizes to engage in two-sided combat whether in a simulator or using 
actual equipment on an instrumented range. Interoperability in DIS is established through 
the use of standard protocols, communication architectures, interface standards, and other 
standardized elements necessary to link the various simulators into a seamless synthetic 
environment. [Ref. 12: p. 1-7] 

With DIS, the military will not only retain the use of stand alone simulators, 
but will be able to conduct "Total Force" training without requiring units to leave their home 
bases. As such, a cost savings will be gained from avoiding the transportation of personnel 


and equipment that would normally be required to facilitate an operation and the reduction 





of wear and tear on the equipment. Additionally, DIS provides a safe environment where 
operation and contingency plans can be developed, tested, and evaluated in a realistic 
simulated environment. Fielding of deployable simulators aboard ship and at forward 
locations, and the use of satellite communications, will enable exercising Marine Air-Ground 
Task Forces (MAGTF) though the use of simulation. Feedback from real simulations into 
virtual simulators will be accomplished through the continued development of instrumented 
ranges that instrument personnel, weapons, and targets to provide data on events in both time 
and space, between opposing sides in simulated combat. [Ref. 7: Chap. 1, p. 5] 

If real time interaction between simulated crewed platforms is to be achieved, 
many issues must be resolved that are not addressed in the commercial market. To enable 
DIS to operate in real time, a dedicated, high capacity communication network with transfer 
rates higher than 155 megabytes/second will be required, and must include land sea and air 
connectivity [Ref. 7: p. 3-35]. The network must be capable of conveying data to interested 
simulation nodes with low latency of 100-300 milliseconds and low latency variance. The 
latency standard refers to the maximum allowed time for data to travel from the transmitting 
application to the receiving application. DIS requires that each simulated entity inform all 
other simulated entities of its activities without knowing who or what the capabilities of the 
other entities are. To handle this many to many communication, the network must support 
multicast addressing rather than point to point addressing. The Defense Advanced Research 
Projects Agency (DARPA) has sponsored the development of the Defense Simulation 
Internet (DSI) as a high capacity, general purpose, packet switching Wide Area Network 
(WAN) to achieve this goal. Another challenge is handling the immense amount of data 
arriving at each node and separating only the data pertinent to that node from the data stream 
for processing at that node. [Ref 12: p. 23-29] 

Technological developments critical to simulation and training that are 
associated with DIS include networks, semi-autonomous forces (SAFOR), terrain and 
environment databases and display, range instrumentation, and individual combatants [Ref. 
1: Chap. 6, p. 2-19]. Infrastructure developments required to support these efforts include 


developing hardware suites for data processing and display for the instrumented ranges and 





developing hardware suites for mission preview capability [Ref. 6: p. 3-9]. User interface 
development for virtual simulation will focus on three dimensional displays, SAFOR that 
correctly model personnel and weapon capabilities, and SAFOR construction tools capable 
of rapidly building synthetic environments to promote rapid scenario and mission generation. 


[Ref. 7: Sec.2, p. 9-17] 


D. THESIS OUTLINE 

Chapter II, Military Simulation Systems, provides an overview of military 
simulations efforts within both the DoD and the U.S. Marine Corps. Major modeling and 
simulation (M&S) initiatives from both organizations are discussed to provide a background 
on the major simulation developments, challenges, and objectives. Finally, the LeatherNet 
project is described to provide an overview of its purpose, the development strategy, and the 
expected benefits of the system. 

In Chapter III, the methods for conducting training and cost effectiveness analysis 
of simulation systems are examined. The challenges to performing such analysis are 
discussed to provide an explanation of how the methodology for evaluating LeatherNet 
was developed. 

Chapter IV, Methodology, explains the purpose and methodology for evaluating 
LeatherNet. The methodology for data collection and analysis is described for both the user 
survey and the user performance evaluation. The test constraints and environment, to 
include the threats to the validity of the experiment, are also examined. 

The results of the data analysis are discussed in Chapter V. The results are displayed 
graphically in bar charts and further explained in the text. Various data analysis techniques 
are used to analyze the user survey and user performance evaluation results. 

In the final chapter, Conclusions and Recommendations, the meaning of the data 
analysis results and their implication on the LeatherNet system are discussed. 
Recommendations for future research in evaluating LeatherNet and similar simulation 
systems that are used for collective training are presented. Additionally, suggestions are 


made regarding steps the Marine Corps can take to promote the use of M&S technologies 
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and to ensure the Marine Corps remains a relevant participant in M&S development. 
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Il. MILITARY SIMULATION SYSTEMS 


A. ASSESSMENT OF MODELING AND SIMULATION IN DOD 

1. Background 

The M&S developments prior to 1990 were characterized by fragmentation and 
limited coordination across the Services and functional communities. In recognition of these 
deficiencies, the DoD created the Defense Simulation Management Office (DMSO) to 
facilitate the coordination of M&S activities within and across the different communities. 
Additionally, many of the DoD components established their own M&S organizations to 
coordinate their efforts within their own component and with the other services. These steps 
have fostered advances in architecture, standards, and protocols; representation of the 
environment, weapons systems, and human behavior; and the fielding of M&S systems and 
associated infrastructure. [Ref. 13: Chap. 3] 

2. Architectures, Standards, and Protocols 

Interoperability and reuse of simulation systems are limited within the DoD due to 
the lack of a common framework for simulation architectures. It is widely recognized that 
a common framework must be established to facilitate the interoperability of all the various 
simulators. To meet this objective, DIS protocols and standards have been implemented to 
establish a common data exchange environment, and they continue to be refined. There 
remains the need to significantly expand DIS and develop its architecture to support a 
broader range of capabilities such as representing command and control more realistically, 
supporting simulations with different time management methods, and reducting the 
computational and communication bandwidth demands. 

The Aggregate Level Simulation Protocol (ALSP) confederation of models currently 
in use will remain the primary tool for simulation for joint level training until the Joint 
Simulation System (JSIMS) reaches its Initial Operating Capability TOC). ALSP has been 
critical for constructing "federations" of existing models for connecting theater-level tactical 
simulators supporting a wide spectrum of joint and combined exercises, such as 


REFORGER. This confederation of models, however, has only limited interoperability, takes 
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a long time to set up, and requires many personnel to operate. DIS, as a part of JSIMS, will 
replace ALSP. [Ref. 14: p. 1147] 

3: Representation of the Environment, Systems, and Human Behavior 

The simulated terrain representations achieved to date are impressive, but they are 
resource intensive to produce and are largely non-reusable by different simulators. The DoD 
continues to conduct research and development for reusable terrain and environmental 
databases, but is impeded by a lack of clear understanding of the modeling and simulation 
community's requirements for environmental data. 

The DoD is also researching the development of authoritative models for 
representing military systems and units as a means of enhancing interoperability and reuse. 
Efforts are underway to develop a joint simulation object library and develop common 
approaches for representing threat forces and systems. Currently, however, there are no 
broadly accepted standards to represent military systems and units, which causes 
incompatibility among the different models. 

The representation of friendly and threat humans and groups of humans under the 
stress of combat is an extremely challenging task. The U.S. Army Modular Semi-Automated 
Forces (ModSAF) program is attempting to provide a baseline, standardized, modular 
software structure in which the model components have a well-documented and defined 
interface. This program is seeking to develop more sophisticated, generalized representations 
of behaviors, missions, and behavior control mechanisms. Other efforts are focused on 
representing command and control in entity-based simulations, but there are few efforts, to 
date, to represent tactical human behavior authoritatively, particularly threat forces, their 
doctrine, and tactics. 

4. Fielding M&S Systems and Associated Infrastructure 

To fully realize the benefits of widespread modeling and simulation use, attention 
must be paid to fielding M&S systems that are interoperable, reusable, support operational 
needs, and address the full range of defense missions. The fielding must be in adequate 
numbers to meet DoD-wide end user needs. To be accepted, modeling and simulation needs 


to have established credibility. Procedures for verification, validation, and accreditation 
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(VV&A) need to better defined and refined. Furthermore, VV&A must begin much earlier 
in model development and become a part of the M&S life cycle. A reusable communications 
infrastructure with the capacity to adequately support modeling and simulation does not exist 
but efforts are ongoing in this area. The DSI has been implemented to support geographically 
distributed users. Plans also exist to merge DSI with the Defense Research and Engineering 
Network (DREN) to enhance capability and reliability. The long term objective 1s to use 
commercial services and operational communications capabilities to meet M&S needs, but 
challenges such as bandwidth reduction and security must be met to make this objective 
feasible. 

In summary, modeling and simulation today is resource intensive and tends to be 
comprised of narrowly focused systems that cannot interoperate among user communities, 
and the systems that do exist have not been properly validated, verified, and accredited. The 
issues are being addressed and significant effort is being made to implement DoD-wide 
architecture, standards, and protocols to promote interoperability and software reuse. [Ref. 


13: Chap 3] 


B. DOD MODELING AND SIMULATION OBJECTIVES 

The DoD has adopted six modeling and simulation objectives to improve its ability 
to more efficiently develop simulation systems and promote simulator interoperability. The 
objectives are as follows: 

@ Provide a common modeling and simulation architecture 

@ Provide timely and authoritative environmental representations 

@ Provide authoritative representations of systems 


@ Provide authoritative representations of human behavior 


® Provide a modeling and simulation infrastructure to meet the developer and end 
user needs 


@ Share modeling and simulation benefits with other government agencies, industry, 
and allied nations [Ref. 15] 
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C. DOD INITIATIVES 
1. SIMNET and CCTT 
In 1983, the U.S. Army and DARPA began the development of the SIMNET 


training system to link various units over large scale computer networks in a common 


synthetic environment. Numerous simulators configured around a mock up of the interior 
of an armored vehicle allowed the participants to see a simplistic three-dimensional 
landscape and other simulated vehicles participating in the exercise. The simulated armored 
vehicles could maneuver over the simulated terrain and engage enemy targets in concert with 
other friendly forces. SIMNET made use of SAFOR, both friendly and enemy, that 
performed automatically according to specified doctrine and tactics after being tasked by the 
computer operator. Use of SAFOR added realism and computer generated forces without 
the use of additional simulators that were normally required to generate and control a 
simulated entity. [Ref. 16: p. 10-12] 

For those tasks that are fully represented in SIMNET, training with SIMNET is more 
effective than additional field training. SIMNET has the capability to train tank crews in 
35% of the tasks specified in the Army Training Evaluation Program and many studies have 
shown a positive transfer of tactical training with SIMNET to field training [Ref 9]. 
Research has shown that platoons trained with SIMNET improved there scores on 
Situational Training Exercises, designed to evaluate a tank platoon in both individual and 
collective tasks, by an average of 13% while platoons receiving only addition field training, 
| improved there scores by only 6%. Additionally, platoons with more battle runs on SIMNET 
produce higher scores in competition for the Canadian Armor Trophy. Although SIMNET 
has been assessed as providing positive training benefits, those benefits have never been 
fully quantified. Tests of the system have tended to be subjective, failed to use control 
groups, and have even been altered as a cost saving measure due to the high cost of testing. 
(Ref. 9] 

Several limitations of SIMNET, which included unrealistic behaviors and modeling 


only two ground combat vehicles, led to the development of the Close Combat Tactical 
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Trainer (CCTT). The design for CCTT began in 1992 as a simulator for training armor and 
mechanized forces. CCTT is the first in a series of the envisioned Combined Arms Tactical 
Trainer (CATT) systems which will eventually include simulators to train artillery, aviation, 
air defense, and engineer forces. 

A task force of the Defense Science Board found that computer-based simulators 
offer the only practical and affordable means for joint commanders and staffs to exercise 
decision making skills, test war plans, and train as a joint force. Despite this finding in 1988, 
uncertainty remains regarding the training effectiveness of simulation and how to properly 
integrate computer-based simulation and field training. While technical tests provide for 
validation and verification as to whether CCTT can meet system requirements, user testing 
is necessary to evaluate the training effectiveness of CCTT and its impact on mission 
performance. To accomplish user testing, the Army plans the most expensive and 
comprehensive test of a simulation system to date, beginning late in 1996. Units will 
undergo pre-training evaluations, training with CCTT, and post-training evaluations that 
include similar training exercises to evaluate the impact of CCTT training on combat 
effectiveness and survivability. To determine the proper mix of simulation and field 
exercises, different test groups will receive different levels of CCTT training, ranging from 
0 - 6 days, mixed with 4 - 10 days field training for a total of ten days training for each 
group. One group will receive no training as a control group. The test is designed to compare 
the training effectiveness of different levels of simulation training to conventional field 
training. [Ref. 16: p. 14-38] 

2. Joint Simulation System 

In the past, each of the Services has developed independent modeling and simulation 
systems that cannot freely interact with systems developed by the other Services. As a short 
term fix, linking of the various models has been achieved through the use of the multi- 
service Aggregate Level Simulation Protocol (ALSP). However, ALSP is a simulation 
protocol with known limitations and inefficiencies that make it unsuitable for long term use 
[Ref. 17]. 


In an effort to better integrate the range of missions of the Armed Forces into a 
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common framework, the DoD will migrate to the JSIMS as a replacement for the ALSP 
confederation of models. The impetus for JSIMS development is to eliminate the individual 
services duplication of effort in modeling and simulation development and the resulting 
interface difficulties, data sharing problems, and inefficient use of funds [Ref. 14]. JSIMS 
will provide integration of constructive, live, and virtual modeling and simulation 
capabilities, with C4I fully supported and interfaced using real-world equipment. The system 
will include man-in-the-loop capability with incorporation of decisions and simulation at 
all levels, including the ability to train down to the individual level. [Ref. 17] 

Funding and development of JSIMS will be a cooperative effort with participation 
from all four Services. The Marine Corps will participate in this effort by contributing 10% 
of the JSIMS core funding with each of the other services contributing 30%. Besides 
providing funding, the Services will identify and nominate the best service applications and 
algorithms for use in the common framework. The Services will ensure a common, 
economical, and efficient development process for their prototypes. The primary 
development goals are reusability, portability and interoperability of obj ect-based software 
components while ensuring DIS compatibility. All the Services have signed a Memorandum 
of Agreement outlining their commitment to JSIMS development, although the Army 
continues to develop their next generation model, WARSIM 2000, which parallels the 
JSIMS effort. 

JSIMS development will initially focus on the training environment at the campaign 

level. Initial operational capability for Joint Task Force level training is scheduled for 1999, 
and by 2003, JSIMS is scheduled to have full operational capability for Service applications. 
[Ref. 18] 


D. MARINE CORPS MODELING AND SIMULATION EFFORTS 

1. Initial Efforts 

The Marine Corps recognizes that the advancement in computer and M&S 
technologies, such as DIS, offers significant benefits in terms of improved training 


effectiveness and cost reduction. The ability to integrate various models and simulators, that 
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are geographically distributed, in a common synthetic environment, offers the opportunity 
for the Marine Corps to conduct Total Force training without the requirement for units to 
leave their home bases. Commands can conduct mission planning, rehearsal, and debriefs, 
and test operational and contingency plans in a synthetic environment with less cost, time, 
and effort. DIS also provides for the development of virtual prototyping of new systems and 
refinement of their designs before the investment of resources in the physical prototypes. 
Similarly, combat developers can harness DIS to rapidly and economically explore new 
operational concepts and doctrines. 

To promote the employment of M&S technologies and to take advantage of these 
promising benefits, the Marine Corps has taken several steps. The most significant steps 


include: 


e Establishment of the Marine Corps Modeling and Simulation Management Office 
(MCMSMO) as the Marine Corps focal point for modeling and simulation 


@ Establishment of an M&S organizational structure that parallels and interacts with 
the DoD M&S management structure 


@ Active participation in joint M&S initiatives 


® Development and implementation of the Marine Air-Ground Task Force 
(MAGTF) Tactical Warfare Simulator (MTWS) for battle staff training 


e Establishment of an Advanced Distributed Simulation (ADS) demonstration site 
at MCAGCC to support the Marine Corps participation in the DARPA's 
Synthetic Theater Of War (STOW) project 


e Establishment of an ADS demonstration site and Decision making Support Center 
(DMSC) at Quantico, Virginia, for experimentation in the use of M&S to support 
analysis efforts 


@ Implementation of a M&S Master Plan and a M&S Investment Strategy [Ref. 7: 
p. 1-5] 


The following sections will provide an overview of the Marine Corps modeling and 


simulation vision and some of the current efforts to turn that vision into a reality. 
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E. MARINE CORPS MODELING AND SIMULATION VISION 

1. Overview 

The Marine Corps M&S vision incorporates two orientations. The first is internal, 
focusing on the use of M&S within the Marine Corps to improve mission performance across 
the Total Force. The second orientation is external and focuses on Marine Corps 
participation in the DoD M&S initiatives. 

The Marine Corps will leverage emerging M&S technologies, such as DIS, to replace 
and/or incorporate the current stand alone models. The objective is to augment traditional 
live fire training at all levels through the integration of combat systems, command and 
control nodes, trainers, logistics and support systems, and constructive models and war 
games into a common synthetic environment. Linking of the various individual, crew, and 
small unit simulators into a common synthetic environment will support MAGTF Battle 
Staff training at all levels as well as joint-level training. The end result will permit the Total 
Force to train as it fights whether in garrison, deployed aboard ship, or deployed in other 
forward areas. [Ref. 7: p. 1-9] 

2. Envisioned M&S Endstates 

To maximize the full potential of M&S technology, connectivity will play a critical 
role in the realizing the Marine Corps M&S vision. The various models and simulators used 
at all levels of training require interconnectivity not only within the Marine Corps, but in the 
joint environment as well. Developing this network of simulators on a global scale will foster 
Marine Corps participation in joint training and planning exercises and permits a role in 
development and evaluation of joint tactics and doctrine. 

M&S is additionally envisioned as playing a critical role in the Marine Corps Combat 
Development Process. Simulated environments will be used to develop concepts, validate 
requirements, and conduct course of action (COA) analysis by exercising force on force and 
warrior-in-the-loop simulations. Acquisition costs can be reduced through the use of virtual 
prototyping that will replace the need for expensive physical prototyping to design and test 
combat system effectiveness. System designs will be refined prior to incurring the cost of 


physical development with the use of simulated environments to evaluate design alternatives 
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and analyze the performance tradeoffs. As system development matures, simulated 
environments will be used to evaluate and revalidate the system performance prior to full- 
scale development and production. 

In an effort to guide the attainment of the Marine Corps M&S vision and to focus 
Marine Corps efforts, a set of eight M&S endstates have been laid out in the Marine Corps 


Modeling and Simulation Master Plan. These endstates are as follows: 


e Exercise any size Total Force MAGTF as part of a combined or joint force from 
home bases, aboard ship, or forward deployed through the seamless integration 
of live, virtual, and constructive simulations 


® Conduct mission planning in a distributed environment 


® Conduct mission preview and rehearsal on land or at sea at all levels, from the 
individual Marine to a Marine Expeditionary Force (MEF), within 48 hours of 
tasking 


@ Validate Marine Corps requirements and doctrine using M&S as a primary tool 


e Participate in the fundamental improvement of the acquisition process by 
simulating before "we buy, build, or fight" 


® Merge M&S and command, control, and communications systems 


e Support every major weapon system in the Marine Corps with a simulator that 
can be networked into a common synthetic environment 


@ Use M&S as a primary decision support tool [Ref. 7: p. 1-12] 


F, TEAM TARGET ENGAGEMENT SYSTEM 

The Team Target Engagement System (TTES) is a Marine Corps sponsored 
advanced technology demonstration to develop a trainer for individuals and small units. This 
simulator will enable users to participate in synthetic force on force combat in an urban 
environment and to engage interactively with Computer Controlled Hostiles (CCH). The 
objective is to provide the users with tactical realism and hostile reactivity to friendly force 
actions while avoiding the cost and use of human instructors and other simulator stations to 


control the simulation of other participants in the exercise. 
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The focus of CCH development is to create hostile entities that are realistic 
participants in the simulation by designing them to move exactly as humans do in an urban 
combat environment, suffer wounds when hit, and "see" users when they are visible and 
react accordingly. To accomplish this, the development must take into account the opposing 
force mission, available resources, acceptable losses, characteristics of the terrain and the 
structures, as well as the actions of the users. To achieve the objective of developing entities 
that will operate autonomously without the control of an operator, the individual human 
represented by the system will consist of a physical model level, a control level, a reactive 
decision level, and a reasoning level. [Ref. 19] 

The physical level will describe human movement, action capabilities, simple models 
of weapons use, performance degradation from wounds, and audio and visual perception. 
The reactive decision level selects appropriate action for the immediate situation and is to 
be designed to have minimal computation at this level so that the decision process can be 
repeated frequently. The reactive decision level will also be designed to incorporate planning 
capabilities several seconds to several minutes into the future to provide a more realistic 
hostile response to friendly actions. Finally, the control level serves to govern movement and 
continuous low level processes. 

In addition to developing the individual human entities, a major objective of TTES 
is developing a semantically rich representation of buildings. In doing so, it will allow the 
computer generated forces to reason about such issues as movement, routes, cover and 
concealment, and firing positions by providing information about walls, apertures, rooms, 
stairs, and other urban features. 

TTES will be designed to be DIS compliant so that these simulators can interact with 
other simulators such as LeatherNet. However, since current DIS protocols do not allow 
simulation of individual combatants, the TTES project must propose extensions to the 


existing protocols. [Ref. 19] 
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G. LEATHERNET 

1. Background 

The advancement of computer and visualization technologies, specifically virtual 
visualization of combat scenarios, has led DARPA to apply these technologies toward the 
development of training and scenario analysis tools. To demonstrate the capabilities of these 
tools, DARPA has planned STOW-97 as a proof-of-principal demonstration with a large 
scale joint exercise using live, virtual, and constructive forces. The Defense Simulation 
Internet will be used to connect these forces comprising 50,000 entities. STOW-97 will 
provide a synthetic environment in which human users can interactively conduct realistic 
wartime scenarios involving friendly and enemy forces, with the ability to review and 
debrief the events, and conduct "what if" analysis. [Ref. 20: p. 1-4] 

2. Overview 

LeatherNet is a virtual simulation system that is being developed at MCAGCC, 29 
Palms, California, under the sponsorship of DARPA. The LeatherNet project provides 
DARPA the opportunity to develop U.S. Marine Corps Semi-Automated Forces (MCSAF) 
for use in the STOW-97 demonstration. DARPA's objective is to capture the unique features 
and behaviors of Marine Corps systems. The objective requires development, in a distributed 
model, of the arbitration of battle outcomes at the entity level of detail to include portrayal 
of both individual weapons systems and individual combatants. Additionally, efforts are 
being taken to provide a realistic representation of Marine Corps command nodes, both 
ashore and afloat, which will portray the influences of one command level over the actions 
of another. The end product of the LeatherNet project is intended to represent a MEF in 
ground maneuver warfare, to include amphibious functions, with all the weapons systems 
and the various positions in the MEF organization in which humans function. [Ref. 20] 

3; Project Strategy 

LeatherNet development is being implemented in four builds. The first two builds 
were conducted during FY 1994 and 1995 and builds three and four will be completed 
during FYs 1996 and 1997 respectively. A brief summarization of the builds follows: 


@ Build 1 Consists of developing a capability to conduct a "terrain walk" of 
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Range 400 at MCAGCC, 29 Palms and the ability for a Company Command 
Element and subordinate leaders to conduct mission planning, briefing, rehearsal, 
and debrief for this range in a simulated environment. 


e Build 2 Implements a capability for Company Command Elements and 
subordinate leaders to conduct a "terrain walk" of the Delta Corridor at 29 Palms 
as well as conduct mission planning, briefing, rehearsal, and debriefing for this 
training area in a simulated environment. 


© Build3 Implements the same capabilities as in Build 2 except that development 
is focused on the Battalion Landing Team Command Element and subordinate 


leaders. 


© Build 4 Implements the same capabilities as in Build 2 except that development 
is focused on the Regimental Landing Team Command Element and subordinate 


leaders. 


Development of LeatherNet is dependent upon the Tactical Exercise Evaluation and 
Control Group (TEECG) instructional staff at 29 Palms to provide feedback to ensure that 
the existing Range 400 and Delta Corridor training scenarios are accurately represented by 
the system. [Ref. 20: p. 11-16] 

4, Benefits of LeatherNet 

The envisioned benefits of LeatherNet are to provide the Marine Corps with lower 
cost and more effective training. These benefits will be derived through the capability of 
studying different tactical alternatives and conducting "what if" analysis without the use of 
field exercises. LeatherNet will enable units to conduct large-scale exercises against a hostile 
force without risk of casualties, expenditure of ammunition and fuel, or wear and tear on 
equipment. Other benefits of virtual simulation, as previously discussed, can be gained from 
LeatherNet's implementation as well. These benefits may be enhanced by providing 


LeatherNet capability to other locations in addition to the facility at MCAGCC. 
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Il. EVALUATION OF SIMULATION SYSTEMS 


A. VALIDATION AND VERIFICATION 

To establish the credibility of M&S systems, the DoD is reliant upon validation and 
verification techniques. The validation process or the "process of determining the degree to 
which a model is an accurate representation of the real world from the perspective of the 
intended uses of the model" and the verification process or the "process of determining that 
a model's implementation accurately represents the developers conceptual description and 
specification" have never been agreed upon [Ref. 21]. The techniques and how they should 
be applied to establish the credibility of the M&S system when used for a particular 
application is the subject of much debate. Many techniques exist for the developer to 
establish validation and verification to include mathematical proofs, design walk throughs, 
and code verification to name a few. However, the basic questions to be answered are; "What 
do we need the M&S system to do?" and "How well can the M&S system do them?" The end 
result must be acceptance by the end users of the M&S system. To reduce the risk of 
executing a M&S program that fails to meet requirements, a review of the M&S system in 
operation by subject matter experts (SME) as early as possible in the development is 
essential [Ref. 21]. Determining whether a system provides the required functions and 
whether those functions accurately model the real world are only some of the considerations 
that should be taken into account in fielding a military M&S system. The training 
effectiveness and cost of the system are other prominent considerations that must be 


considered. 


B. TRAINING EFFECTIVENESS ANALYSIS 

There are no well defined methods for determining military value of a training 
program or system, but military value is ultimately reflected in the degree of combat success 
that any training program or system is responsible for. Unfortunately, the degree to which 
a training program or system enables success in combat is difficult to assess. Military value 


can be empirically determined only in combat and it is obviously impractical and unwise to 
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wait for war to determine the effectiveness of a training system. [Ref. 22: p. 7] 

As a more practical approach to assess training effectiveness, measures of 
effectiveness (MOE) are used as an alternative to actual combat to predict combat success. 
MOE are equivalent to dependent variables that are used to assess the impact of an 
experimental treatment condition [Ref. 22: p. 7]. MOE can be defined as "tools that assist 
in discriminating among a number of alternatives." They show how the alternatives compare 
in meeting functional objectives and mission needs. [Ref. 23: p. 8-7] 

In selecting MOE, a distinction can be made between objective and subjective data. 
Objective data are based on observable events whose occurrence is not usually the subject 
of dispute. Examples include a student's test score or the number of hits scored on a gunnery 
range. Subjective data are based on the opinions and judgements of a SME. Subjective data 
is necessary to capture intangible information such as leadership, decisiveness, morale, and 
motivation. While the SME evaluation may carry great weight, it may be neither accurate 
nor valid. [Ref: 22: p. 12-14] 

Selection of training MOE is dependent upon the tasks necessary to accomplish the 

training, the conditions under which the task will be conducted, and the standards for the 

tasks. Evaluation of collective training is much more difficult and subjective than evaluation 
of individual training. The problem is further compounded when attempting to evaluate 
simulators. The Government Accounting Office (GAO) has concluded that despite the 
Army's strategy to increase the use of simulation, the strategy lacks detailed guidance for 
commanders to make effective use of simulators. The major downfall is the lack of linkage 
between the simulation and the tasks the unit can expect to perform in wartime. The lack of 
guidance has resulted in the less than optimal use of simulators. That being the case, the true 
training value may not be realized nor evaluated. [Ref. 24: p. 2] 

Deciding how to measure the training effectiveness of a training program or system 
is neither obvious nor simplistic. The training environment must be considered to determine 
whether the training is structured or unstructured. Schools tend to offer structured training 
and be more objective in nature. Unit training tends to be unstructured and evaluation is 


more difficult and subjective due to reliance upon the opinion and judgement of SME. 
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Further difficulty arises from the lack of understanding of the important dimensions of 
collective training and evaluation. Even developing the task list to determine the MOE is 
complicated by the fact that the tasks tend to exist in hierarchies, and separate task lists can 
be developed at each level in the hierarchy. As the number of levels increase, tasks are added 
to each level for possible interaction between that level and other levels. A comprehensive 
task list would be comprised of all the tasks at each level and all the tasks involving 
interaction among levels. No standard method of developing collective MOE exists. Many 
researchers stress the importance of relying upon SME's to assist in developing MOE. SME's 
enable researchers to identify indicators of success, interpret performance, and to ensure all 
relevant tasks are measured. While the SME's know best how to assess the complex training 
environment, objectivity may be lost as this method is reliant upon opinion and judgement. 
[Ref. 22: p. 10-18] 

In using MOE certain assumptions must be made. In order to conduct a training 
effectiveness analysis of one training method over another, a MOE that predicts combat 
success must be selected. The MOE might be student grades or an evaluator's rating. To use 
these MOE, one must make a transfer assumption that student grades or an evaluation rating 
predict combat success. Such an assumption implies a series of assumptions that school or 
training performance influences training readiness which influences combat readiness. While 
research has provided data supporting the linkage between school performance, field 
performance, and combat readiness levels, these linkages remain largely unconfirmed. [Ref. 


22: p. 13] 


C. COST EFFECTIVENESS ANALYSIS 

Cost effectiveness analysis (CEA) is a method used by the DoD to make decisions 
in selecting alternative courses of action when the results effect military performance. While 
the inputs can be measured on a cost basis, the outputs or military value, cannot be measured 
in monetary terms. CEA is similar to cost benefit analysis in respect to taking into account 
all of the associated costs of a project throughout a projected life cycle. However, CEA takes 


into account military value which cannot be defined in the same terms as cost due to a lack 
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of market value for increased readiness, improved training, and improved weapon systems. 
Since cost and military value have different units of measurement, selection of alternatives 
cannot be made on a cost basis alone. Orlansky developed a decision logic diagram for 
evaluating the cost and effectiveness of two training programs or systems. The diagram is 


shown below: 


EFFECTIVENESS 
LESS SAME MORE 


LESS 


SAME 
MORE 





Table 1. Cost and Effectiveness Decision Diagram [Ref. 1] 


Using the diagram in Table 1, if an alternative is as effective and costs less than 
another, it should be adopted. Likewise, if an alternative is more effective and cost the same 
or less than another alternative it should be adopted. An alternative should be rejected if it 
is less effective and cost the same or more than the other alternative or the alternative is as 
effective and cost more. Finally, in those situations where the alternatives exhibit less 
effectiveness and less cost, equal effectiveness and equal cost, or greater effectiveness and 
greater cost, no logical preference can be made. [Ref. 1: p. III-2] 

While Orlansky recommends acquiring simulators that are equally effective and cost 
less than the actual equipment, he suggests that the military should strive to acquire 
simulators that provide increased effectiveness at the same or lesser cost. He believes that 
the military should take the same approach to acquiring simulators as it does to acqiuring 
weapon systems with the focus on increasing effectiveness. Costs should be a secondary 
consideration. The end result should be the optimum combination of simulators and actual 


equipment that provide the most effective training at the least overall cost. [Ref. 1: p. T-11] 
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In a study by the M&S Benefits Task Force, DMSO reports the training applications 
of M&S for individual skills training are wide spread and the training benefits are positive 
when simulation is properly mixed with training on real equipment. Analyst have well- 
established theories and experimental methods for conducting analysis at this level. The 
same cannot be said for unit training, particularly in the higher echelons. The high cost of 
repeated large scale exercises and the difficulty of conducting the controlled experiments 
necessary preclude the collection of meaningful data. Multi-million dollar savings are 
reported, however, when comparing computer simulated command post exercises with 


conventional field training exercises. [Ref. 9: p. 2] 
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IV. METHODOLOGY 


A. BACKGROUND 

1. Purpose 

The purpose of this research is to evaluate the perceived ability of LeatherNet to meet 
the mission planning, and to some extent, the mission preview objectives of the Marine 
Corps Modeling and Simulation endstates as described in the Marine Corps Modeling and 
Simulation Master Plan. Secondly, this research will determine if a correlation exists 
between the exposure to conducting mission planning and briefing with LeatherNet, and 
subsequent user performance in conducting live fire training. 

Zi Test Constraints 

User testing and evaluation of LeatherNet has been constrained by several factors. 
All aspects of LeatherNet could not be evaluated since the system remains under 
development. The number of test participants has been limited by the number of units 
participating in the Combined Arms Exercise (CAX) Program, which are the units 
designated by the Commanding General of MCAGCC to conduct user testing of LeatherNet. 
The CAX program is an ongoing Marine Corps program to train Marine infantry battalions 
in all aspects of fire support coordination in a live fire and maneuver environment. CAX 
training cycles last approximately 30 days and are comprised of intense field training in fire 
support techniques and tactics at all levels of command. The training schedule allows units 
little or no opportunity to conduct training other than that training related directly to the 
CAX program. Using other, less encumbered, units to conduct the testing is not practical due 
to the cost of transportation to the remote test site and the cost of the training on Range 400 
in terms of ammunition expenditure. During the period from April to August 1996, a total 
of eight groups were exposed to the system. The amount of time the users were exposed to 
the system was constrained by the demanding CAX training schedule that limited their 
exposure to three to four hours between field exercises. The time constraints further limited 
the functions which could be demonstrated, eliminating those which were time intensive. 


Some functions were not demonstrated due to incorrect behaviors and inaccurate 
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representation of certain entities at this stage of development. For example, hostile infantry 
would stand on the trench lines, rather than in them, rendering them easier to "kill." In these 
instances, the TEECG personnel, who are responsible for conducting the CAX program, felt 
the system provided "negative training" due to incorrect behavior of these entities. The 
TEECG thought such behavior gives a false impression of the difficulty in overcoming 
fortified defenses and decided against testing the system using the full wargaming 
capabilities until the represented behaviors become more realistic with further development. 
3. Test Environment 

a. User Testing Participants 

The officer responsible for coordinating and conducting user testing of 
LeatherNet is Major Daniel Newell, the TEECG LeatherNet Representative. Major Newell 
ensures that two of the three rifle companies from each infantry battalion participating in the 
CAX program receives its Range 400 exercise brief using LeatherNet. The third company, 
which is the control group, is not exposed to LeatherNet and receives its brief through the 
use of a map study. The test participants typically consist of an infantry company's 
Command Element and the Platoon Leaders. The companys have the option of including 
other individuals in the briefs such as their Forward Observers and Machine Gun Section 
Leaders at their own discretion. A total of 31 participants were surveyed. 

The participants come from various Marine Corps bases around the U.S. and 
include Marines from both reserve and active duty units. The participants had varying 
degrees of experience in their Military Occupational Specialty, to include prior exercises on 
Range 400, but none had used LeatherNet prior to the testing. 

b. Functions Evaluated 

As previously noted, due to the maturity of LeatherNet development, time 
constraints, and the willingness of test personnel to test the system, only the mission 
planning and mission briefing functions of the Marine Corps objectives for LeatherNet are 
evaluated. The system, while mature enough to conduct a mission rehearsal against an 
interactive hostile force, and perform "what if’ COA analysis, was not used for these 


purposes during the course of this research. The LeatherNet mission planning and briefing 
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evaluation focused on using the system to conduct an estimate of the situation to include 
taking into consideration such elements as mission, enemy, troops and fire support 
available, time, space, and logistics. The test participants were led through the estimate of 
the situation analysis by Major Newell as part of the TEECG's Range 400 exercise brief 
During the brief, the participants were allowed to use LeatherNet to "navigate" the Range 
400 terrain to conduct a reconnaissance from both the friendly and enemy perspectives. In 
doing so, the participants were able to assess their plan's vulnerability from potential enemy 
positions and weapons systems along their planned maneuver routes by using LeatherNet's 
display of weapons range fans and deadspace. Likewise, using the same functions, they were 
able to evaluate the effective target coverage of friendly weapon systems from potential 
friendly firing positions. Additionally, the "terrain walk" of Range 400 enabled the 
participants to "see" the terrain, the potential enemy positions, and take into account time 
and space consideration for their fire support and maneuver elements. 

User surveys, to measure user opinion of LeatherNet, were conducted 
immediately after the completion of the Range 400 field exercise to ensure minimal 
information loss by the participants due to a time lag between the event and the survey. The 
user survey can be found in Appendix A. 

4. Range Performance Evaluation 

The analysis conducted in this study is based on a comparative evaluation between 
the experimental groups, those units given a mission brief using LeatherNet prior to 
conducting the Range 400 exercise, and the control groups, those units given a mission brief 
using only a map and aerial photo study. The comparison of the two groups is made by 
comparing the percentage of the 31 training objectives for Range 400, as developed by the 
TEECG staff, that are successfully accomplished by each group. The Range 400 evaluation 
worksheet is included in Appendix B. Assigning scores to the evaluation worksheet was 
performed by a single SME, Major Newell of the TEECG. The best alternative training 
method is determined by the alternative with the highest average percentage of training 
objectives successfully accomplished. In some instances, not all 31 objectives could be 


measured due to the absence of certain sub-units conducting the exercise. For example, the 
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absence of Dragon missile teams precludes their evaluation. In these instances, the total 
score for the unit was based on the percentage of remaining objective successfully 
accomplished. The number of training objectives evaluated for the eight experimental and 
four control groups ranged from 28 to 31. 

If the experimental groups demonstrate a higher achievement of training objectives 
to a Statistically significant degree, it can be assumed a positive transfer of military value has 
occurred. However, nothing can be inferred about the transfer as a function of the amount 
of exposure to LeatherNet. To do so would require several different units to experience 
LeatherNet exposures of different time lengths and subsequent evaluation on the range. [Ref 
22: p. 30] 

2: Threats to Validity 

Validity refers to the extent to which the user surveys and performance evaluations 
actually measure the true benefits the users derive from exposure to LeatherNet. To answer 
the basic research questions, numerous investigate questions were considered by the author. 
These questions were pretested to eliminate ambiguous questions and to detect other 
weaknesses in the survey instrument, and refined into the questions found on the survey 
included in Appendix A. To enhance content validity, a review of the user survey by SME 
within TEECG was conducted to ensure adequate coverage of the questions under study. The 
performance evaluation forms for Range 400 used in this study were developed by TEECG 
prior to this study and are considered to be adequate to measure performance since they have 
been developed by the SME responsible for evaluating Range 400 exercises as part of their 
official duties. 

The study consists of a post-test only control group experiment. No performance 
pretest was conducted with either group. The control and experimental groups, consisting 
of infantry company command elements and subordinate platoon and section leaders, were 
determined by random selection. In a post-test only control group study, normally the 
internal threats to validity from history, maturation, selection, and statistical regression are 
adequately controlled by random assignment of groups. [Ref. 25: p. 359-364] 


Despite efforts to promote internal validity, the test environment experienced during 
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this research presented certain threats to the validity of the experiment used to measure the 
benefits of LeatherNet. Ideally, neither group would be exposed to any information source 
concerning the Range 400 exercise than that which is normally available to company-sized 
units prior to an attack upon enemy held territory. Typically, this information would consist 
only of intelligence summaries from higher headquarters, aerial photos, and map studies. 
Both groups received all of these types of information with the experimental group receiving 
additional information available through the use of LeatherNet. There were, however, no 
effective means to control access to information sources not normally available in a hostile 
environment. Prior to conducting the exercise, units could, and in many instances did, gain 
additional information through physical reconnaissance of Range 400, observing other units 
conducting the Range 400 exercise, and receiving advice from individuals and units that had 
previously run the range. If the control units chose to exploit these sources of information, 
it could negate the ability to measure the benefits of using LeatherNet since this information 
is similar to the information LeatherNet provides. 

The manner in which the TEECG conducts its mission briefs for all units prior to the 
exercise effectively reduces the possibility of critical errors by the exercising units, and 
therefore, tends to diminish disparity that might normally be observed between the exercise 
and control groups. The TEECG's purpose is to assist and evaluate unit training and their 
efforts are often focused on helping units train correctly, rather than allowing units to fail 
due to lack of experience and simply giving them an evaluation of their shortcomings. As 
such, training exercise briefs often includes instruction that is prescriptive in nature, causing 
units to execute very similar plans with little variation due to the suggestive remarks. For 
example, supporting weapons such as mortars and heavy machine guns are employed from 
essentially the same positions on Range 400 by nearly every unit. The reasons for this are 
not only due to suggestive remarks, but suggestive names for terrain features such as 
Machine Gun Hill. Specific instructions for weapons employment were also given that left 
little room for variance, due to training safety considerations. Under these circumstances, 
the ability of LeatherNet to influence performance tends to be reduced. 


Other influences upon unit performance evaluation that could not be controlled 
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include the amount of ammunition available to the units and the personnel used to conduct 
the evaluation. When ammunition availability is limited, planning is essential to performance 
to ensure that ammunition remains available in adequate amounts at critical moments of the 
exercise to ensure fire support is available. However, the author observed some units 
conducting the range with up to three times the normal allotment of some ammunition types. 
Such abundant supplies of ammunition give the units a performance edge that enables them 
overcome the lack of information and planning prior to the exercise. An internal validity 
threat also exists in the fact that the performance evaluations were conducted by a single 
individual. The evaluator's boredom, experience, anticipation, bias, and fatigue all have the 
potential to influence results. 

It is acknowledged that it is difficult to conduct an experimental test in an operational 
setting. The problems include difficulty in establishing experimental control in manipulating 
the events for experimental purposes. In order to achieve this objective, the cost becomes 
prohibitive as it requires using units that are solely dedicated to the test process, as well as 
the expenditure of large amounts of ammunition to replicate a combat scenario. To do this 
for the single purpose of testing LeatherNet was considered impractical for a system still 


under development. 


B. DATA COLLECTION 

1. Questionnaire Development 

A user survey questionnaire was developed to measure the user's perception of the 
value of the various aspects of LeatherNet as a mission planning and mission briefing tool. 
Many of the specific questions were developed from a questionnaire previously used by 
Major Newell to conduct user opinion surveys regarding the use of LeatherNet to conduct 
an estimate of the situation. This questionnaire was revised, reformatted, and expanded to 
cover other aspects of this research. The user survey was pretested at the Naval 
Postgraduate School using Marine personnel, several of whom had previous Range 400 
experience, and with the TEECG personnel. Pretesting was necessary to ensure the questions 


were understandable, unambiguous, and in an easily readable format. Numerous revisions 
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were necessary before the final version could be made. 

2. Types of Data Collected 

Questions 1 - 34, 39, and 41 - 44 provide ordinal data using five point Likert scales 
to record the respondent's answers. The respondents are asked to rank their degree of 
agreement with the question statements, ranking from strongly disagree to strongly agree. 
Question 38 also provides ordinal data but does so by asking the respondents to rank, by 
order of importance, various potential capabilities of LeatherNet. The remainder of the 
survey, questions 35 -37 and 40, provide nominal data. These questions require the 
respondents to list LeatherNet characteristics that they found most useful, least useful or 
thought should be developed for the system. Nominal data was derived by grouping the 
characteristics that the respondents listed by category. The following subsections describe 
the purpose for the specific questions used in the user survey. 

3: Determine the User's Experience Level with Range 400. 

Questions 1 - 8 ask the user to indicate his professional experience level, the tactical 
proficiency of his unit, and his familiarity with Range 400. The purpose for these questions 
is to test the hypothesis that LeatherNet is most useful as a mission planning tool to those 
personnel whom are the least experienced and familiar with Range 400. In order to obtain 
an overall rating for an individual respondent's experience and familiarity, the various 
questions are weighted according to their relevance to professional experience and Range 
400 familiarity. The weights were assigned through the use of the author’s and the TEECG 
SME’s judgement after using pairwise comparisons between the criteria to determine their 
relative importance as it relates to experience To obtain the overall experience and 
familiarity of the user, the scores of all the questions were summed to provide an aggregate 
experience and familiarity score. The possible scoring range for experience and familiarity 
ranges from 1, for least experienced; to 100, for most experienced. Table 2, on the following 
page, indicates how questions 1-8 of the user survey were weighted according to the 


respondent's answers. 
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Question 1 
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Question 2 
Scores 
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Question 4 


1 
<6 
0 
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Question 8 
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Table 2. Determination of User Experience Through Assignment of Experience 


Scores (Bottom number in each cell) to Possible Responses (Top portion of each cell) 


to Survey Questions 1-8 


The top portion of each cell represents a possible response to each question and the 
bottom number reflects the experience score assigned to the respondent if he selected that 
particular answer. A sum of all responses to questions 1-8 is used to assign an overall 
experience score for each respondent. 

Admittedly, these questions are not all inclusive in determining experience and 


familiarity but represent those factors which SME feel have the greatest degree of influence 
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on experience as it relates to performance on Range 400. Additionally, while this method 
of assigning weights to the experience and familiarity criteria is subjective, it is used due to 
a lack of a suitable alternative. 


4. Determine the Ability of LeatherNet to Conduct an Estimate of the 
Situation 


Questions 9 -23 ask the respondent his opinion of the degree to which LeatherNet 
helped his understanding of the enemy situation, terrain, fire support, time and space issues, 
and logistical considerations. | 

=i Determine the Realism of the LeatherNet Terrain 

The users are asked in question 24 to indicate how closely the LeatherNet terrain 
represents the real terrain on Range 400. 


6. Determine How the Information Provided by LeatherNet is Used by the 
Test Participants 


Questions 25, 26, 41, and 42 ask if critical information was provided by LeatherNet 
and if that information influenced the respondent's plan of attack. Questions 41 and 42 were 
added to the end of the survey because they might be considered "hot button" issues. The 
questions ask the respondent to indicate if they failed to change their plan of attack despite 
the fact that LeatherNet provided information indicating they should do so. The purpose for 
the questions is to gauge the level of trust users place in the information LeatherNet 
provides. 


Te Determine the Potential of LeatherNet to be a Mission Planning and 
Rehearsal Tool 


The respondents are asked, in questions 27 - 30, to indicate the degree to which 
LeatherNet demonstrates the potential to be an effective training tool for Marines to evaluate 
tactical alternatives, conduct mission planning, and mission rehearsal. 


8. Determine the Value of LeatherNet Under Different Circumstances of 
Use 


The respondents are asked their opinion of LeatherNet's value, in questions 31 - 34, 
if they had no other source of information regarding Range 400, if they had the opportunity 
to expose more of their own Marines to the system, and if they were allowed to use 


LeatherNet at their home base prior to CAX. 
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9, Determine the Most and Least Useful Features of LeatherNet and What 
Features and Changes can be Implemented to Make LeatherNet More 
Useful 


The respondents are asked in questions 35 - 37 and 40 to provide written comments 
concerning what features they felt were the most and least useful, what features should be 
added, and what changes should be made to LeatherNet to make the system more useful. 


10. Determine the Ranking, in Order of Importance, of the Perceived 
Abilities of LeatherNet to Overcome the Limitations of Live Fire 
Training 


Question 38 asks the respondent to rank, in order of importance, eight perceived 
abilities of LeatherNet to overcome the limitations of live fire training. The perceived ability 
of LeatherNet to overcome the limitations of live fire training were determined through 
previous research on the system. The perceived limitations of live fire training include lack 
of realistic and interactive enemy forces, limited feedback on weapons employment, safety 
constraints, lack of adequate range facilities and limited ability to reconstruct exercises. [Ref. 
26: p. 48] 

11. Determine the Respondents Attitude Toward LeatherNet 

Questions 43 and 44 ask the respondent to indicate his attitude toward LeatherNet 
prior to using the system and whether his attitude is more positive after exposure. The 
purpose of these questions is to determine if there is an existing bias towards using 
simulators for training and to determine LeatherNet's degree of acceptance among its users. 
Cc. ANALYSIS PLAN 

1. Survey Data 

The data collected through user surveys in most cases are ordinal data. The questions 
in the survey rely upon opinion scales to register the participants level of agreement to 
various statements. To determine the central tendency of all respondents the median value 
is used and is presented in bar graphs to show the various levels of agreement with the 
statements in the questionnaire. The median is the most appropriate measure to determine 
the center for ordinal data and provides resistance to extreme scores. In some cases, the 
percentages of respondents giving particular answers is given to further clarify the 


distribution of responses and levels of agreement. 
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The level of experience, as calculated from survey questions 1 - 8, is further analyzed 
through the use of Spearman's Rho, to determine if there is a correlation between the level 
of experience of a user and the perceived benefits of using LeatherNet. Spearman’s Rho is 
used to test the null hypothesis, that there is no difference between a participant’s experience 
and the degree to which LeatherNet provides critical information and influences the user’ s 
plan of attack [Ref. 25: p. 507-509]. The user's experience score is correlated with his 
responses to questions 25 and 26 to make this determination. 

2. User Performance Data 

Performance scores for the units using LeatherNet and the control units are compared 
to determine if there is a quantitative performance differential between the units using 
LeatherNet to conduct mission planning and the control groups, units with no LeatherNet 
exposure. The comparison of the two groups is made by comparing the percentage of the 
31 training objectives for Range 400, as developed by the TEECG staff, that are successfully 
accomplished by each group. The evaluation sheet for Range 400 is included in Appendix 
B. 

Due to the small sample sizes involved, an n of four for the control group and an n 
of eight for the experimental group, the t test is used to test for a statistically significant 
difference in the mean percentage of training objectives successfully completed by each 
group. The t test is chosen because the data, or performance scores, are interval level data 
and the samples are independent. Use of the t test assumes normally distributed populations 
and equal population variances [Ref. 25: p. 449-451}. This test is designed to test the null 
hypothesis, that there is no difference in performance on the Range 400 exercise for the units 
using LeatherNet as compared to the units without LeatherNet exposure. The alternative 


hypothesis is that using LeatherNet does make a difference in performance. 
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V. DATA ANALYSIS 


A. USER PERCEPTIONS 

1. Description of the Chart Data 

The bar charts used in this chapter use numbers 1 - 5 to represent the user's level of 
agreement with the survey questions. The numbers translate as follows: 1 - Strongly 
Disagree, 2 - Disagree, 3 - Neither Agree nor Disagree, 4 - Agree, 5 - Strongly Agree. 
Some charts do not display bars for all five numbers which indicates that no test participants 
selected the response category represented by the missing number. 


2. Assessment of LeatherNet as a Tool to Conduct an Estimate of the 
Situation. 


The user's perception of using LeatherNet as a tool to conduct an estimate of the 
situation is generally positive. The median value of all responses to survey questions 9 - 23 


are displayed in Chart 1. 
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Chart 1. Assessment of LeatherNet for Conducting an Estimate of the 





Situation 


Of all the elements of conducting an estimate of the situation, the users agreed most 
strongly that LeatherNet improves their understanding of the enemies fields of fire (question 


10) with 67% of all respondents answering “strongly agree.” The bar for question 10, as 
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shown in Chart 1, indicates the median value was 5 The respondents tended to "agree" that 
LeatherNet improved their understanding of most of the other aspects of conducting an 
estimate of the situation, particularly friendly avenues of approach (question 11) , friendly 
fields of fire (question 15), and shift/cease fire points (question 19). For these three aspects, 
over 80% of all test participants agreed or strongly agreed that LeatherNet improved their 
understanding of them. The test participants had a generally neutral opinion regarding an 
improved understanding of obstacles (question 12), enemy avenues of approach for 
counterattacks (question 13), and ammunition requirements (qustion 23) as indicted by the 
median values of three to each of these questions. 

a2 Assessment of LeatherNet Terrain 

When asked if the LeatherNet terrain closely matched the Range 400 terrain 
(question 24), 81% of all the respondents either agreed or strongly agreed that it closely 
matched the real terrain. The median value was 4 or “agree.” Only one respondent disagreed. 
However, it should be noted that several respondents thought that the best way to improve 
LeatherNet would be to refine the terrain resolution to better show the microterrain. Many 
test participants thought that the terrain tended to look "cartoonish" up close and that it was 
not possible to determine the type of ground material such as soil, sand, or rock. 

4. Assessment of LeatherNet Information Usage 

Charts 2-5 show the distribution of the responses to survey questions 25, 26, 41, and 
42. An assessment of whether critical information was provided by LeatherNet will be made 
followed by an assessment of whether that information influenced the final attck plan. The 
questions were in inverse order on the questionnaire which is why question 26 is addressed 
here first. The distribution of responses to question 26 regarding whether LeatherNet 
provides critical information is shown in Chart 2. The fact that 65% of all responses were 
greater than or equal to the median value of four, indicates there is general agreement that 
LeatherNet provided the respondents with critical information that was previously unknown 
to them. The claim that they were able to gain critical information is important, particularly 
in light of the fact that the respondents had ample alternative sources of information 


regarding the exercise. Despite the use of map studies, aerial photo studies, and in many 
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cases, gathering information from fellow commanders who had previously run the range and 
even physically walking the ground prior to the exercise, they were still able to obtain 
previously unknown information from LeatherNet. The reason for the knowledge gain is 
probably due to the systems ability to present visually that information normally obtained 


by one's imagination, such as weapons fire fans. 
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Q26 
Chart 2. Assessment of Whether LeatherNet Provides Critical Information 


The responses to question 25 shown in Chart 3, however, indicate that there was only 
moderate tendency for the additional information provided by LeatherNet to influence the 
final plan of attack.The median value was four, with 52% of respondents claiming 
LeatherNet influenced their plans. The fact that more respondents claimed to have gained 
critical information than changed their plans based on that information may be explained in 
part by the lack of freedom to alter plans due to range regulations and other constraints 
placed upon the participants. An alternate explanation would be some lack of user 


confidence in the system, but responses to questions 41, 42, and 44 fail to support this idea. 
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Q25 
Chart 3. Assessment of LeatherNet’s Influence on the Final Plan of Attack 


The distribution of responses for question 41, shown in Chart 4, indicates some 
respondent discomfort in relying upon LeatherNet to evaluate their plan of attack because 
they were unfamiliar with the system. While the median response of three indicates neither 
comfort nor discomfort as the central tendency, 19% of the respondents indicated they did 
have some degree of discomfort. However, the responses to question 42, shown in Chart 5, 
indicate that nearly all respondents disagree or express neither agreement nor disagreement 
with the statement that they "did not change their plan of attack because they felt they could 
not trust LeatherNet results for use in a real exercise." Only one respondent agreed that he 
did not alter his plan of attack because he felt he could not trust LeatherNet, and none 
strongly agreed. The assumption here could be that, while the system was unfamiliar to the 
users, the presentation of information appears to be authoritative and reliable enough for 
planning purposes. Before reaching this conclusion, however, it would be wise to consider 
that Marine officers may not desire to reveal that they did not change a plan of attack in the 


face of information indicating that that was the proper course of action. 
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Chart 4. Responses to Question 41 Regarding Whether the Respondents Felt 


Uncomfortable Using LeatherNet to Evalauate a Plan of Attack 





Chart 5. Responses to Question 42 Concerning Whether the Respondents Failed 
to Use LeatherNet Information in the Final Plan of Attack Due to Lack of Trust 


in the System 
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Spearman’s Rho was used to determine if a correlation exists between the user’s 
experience level and the degree to which LeatherNet influenced the user’s final attack plan 
and provided previously unknown critical information. Spearman’s Rho is used to test the 
null hypothesis, that there is no relationship between a participant’s experience and the 
degree to which LeatherNet provides critical information and influences the user’s plan of 
attack. It was expected that the alternative hypothesis, that there is a relationship between 
a user’s experience and the degree to which LeatherNet influences the plan of attack and 
provides critical information, would prove to be true. This expectation was based on the 
typical tendency for more experienced personnel to rely upon their own skills and methods 
rather than adopt a planning tool of unknown value. Upon conducting the test, Spearman's 
Rho values of .0287 and -.1302 were calculated for questions 25 and 26 respectively. A 
Spearman's Rho of 0 signifies no correlation between variables, whereas a Spearman's Rho 
of plus or minus 1 signifies total correlation. It can easily be seen that no significant 
correlation exists between a user's experience and LeatherNet's ability to influence the his 
final attack plan and provide critical information to him. Therefore, we fail to reject the null 
hypothesis. In fact, to be 95 % confident that the null hypothesis is rejected only when it is 
indeed false, a P-value of .05 or smaller must be attained. The P-values of the calculated 
Spearman's Rho were .878 and .485 respectively. 

The implication of these findings is that LeatherNet, as used in these tests, has value 
as a mission planning tool to the test participants regardless of their experience level and 
familiarity with Range 400. For training purposes, the indications are that company level 
staffs, of a wide range of experience levels, can improve their understanding of an exercise 
through simulation, and that LeatherNet training adds value to the CAX training program. 

5; Assessment of LeatherNet as a Mission Planning and Rehearsal Tool 

Charts 6-9 on the following pages display the distribution of responses to questions 


27-30 of the user survey. 
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Count 





Q27 


Chart 6. Assessment of LeatherNet’s Potential as an Effective Mission Planning 


Tool 


Count 





Q28 
Chart 7. Assessment of LeatherNet’s Potential to be an Effective Mission Rehearsal 


Tool 
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Chart 8. Assessment of Whether LeatherNet Provides Vital Information in 


Evaluating Tactical Alternatives 


20: 


10: 





Chart 9. Assessment of LeatherNet’s Potential to be an Effective Tool for Training 


Company Commanders and Their Staffs 
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Overall, the test participants believe that LeatherNet demonstrates the potential to 
be an effective mission planning and rehearsal tool, that it is effective for training company 
commanders and their staffs, and that it provides vital information for evaluating tactical 
alternatives. Less than 10% of all respondents expressed a negative or neutral response to 
questions 27, 29, and 30. The highest number of neutral and negative responses, totaling 
25% of all responses, were given to question 28 regarding LeatherNet's potential to be an 
effective mission planning tool. This is probably due to the fact the participants did not 
rehearse their plans with the system, and were therefore only speculating about the systems 
potential in this regard based on the credibility of the other functions they observed. 

6. Assessment of Possible Alternate Uses for LeatherNet 

The respondents answers to questions 31 -34 indicate wide agreement that use of 
LeatherNet at their home bases, with more time to use the system, and with more of their 
personnel exposed to the system, would be beneficial. Furthermore, they agree that 
LeatherNet would be of more value if other sources of information concerning Range 400 
were not available, such as having the ability to physically walk the range and get advice 
from other personnel who had completed a Range 400 exercise. Only 3-6% of all 
respondents disagreed with any of these four statements. 

7h Assessment of the Most and Least Useful Aspects of LeatherNet 

The most useful features of LeatherNet, according to the respondents answers to 
question 36 are listed below. The features are listed in decreasing order of importance as 
determined by the number of respondents listing each feature. The number of respondents 
listing each feature is given in parenthesis. It is important to remember that the users were 


not exposed to the full functionality of the system. 


@ The ability to display both friendly and enemy fields of fire and engagement areas 
(15) 


@ The ability to "walk the terrain" for reconnaissance purposes (12) 
@ The ability to visually display intelligence on the enemy (4) 


@ The ability to display avenues of approach (3) 
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The least useful features according to the respondent's answers to question 37 are as 
follows: 
® Too many overlays and multi-color templates that are confusing (5) 


@ The possibility of depending on simulators for training at the expense of training 
in a real environment (3) 


@ The presentation of too much information, too fast, to be useful (2) 


@ Lack of details on microterrain such as resolution and surface type (2) 


Below are the user's responses to question 35, listing in decreasing order of 
importance, the users opinions on how to improve the system. These opinion are based on 
having only used LeatherNet for mission planning and briefing. 

@ Develop an ability to war game against a realistic enemy ( 7) 


@ Refine the terrain to give more details on the microterrain and represent different 
types of ground material (5) 


Provide more details on the enemy forces (4) 


Expand the terrain database to cover other areas (2) 


Use at other bases (2) 


It should be noted that with the exception of the microterrain issue, LeatherNet 
already provides, or will soon provide, all of the above listed functions. 


8. Determine the Most Important LeatherNet Capabilities to Overcome the 
Perceived Limitations of Live Fire Training 


The respondents ranking, in decreasing order of importance, of LeatherNet's 


perceived ability to overcome the perceived limitations of live fire traming are displayed in 


list below. 
e Ability to war game against a realistic, responsive enemy 
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@ Ability to obtain quantitative feedback on the effectiveness of weapons 
employment 


@ Ability to train on simulated terrain in the abscence of real terrain 
e Ability to reconstruct/replay an exercise for detailed debrief 


e Ability to conduct the same exercise under varying conditions or to modify a plan 
and evaluate the outcome 


@ Ability to conduct "danger close’ fire support 
@ Ability to evaluate a subordinate's performance 


e@ Simulated casualty assessment 


The order was determined by the median value of all respondent rankings. For those 
characteristics with the same median value, the mean value of the respondent's rankings was 
used to determine the order. The ability to war game against a realistic, responsive enemy 
was perceived to be, by a wide margin, the most important characteristic provided by 


LeatherNet that overcome the limitations of live fire training. 


9, Assessment of the Respondent's Attitude Towards LeatherNet 

Question 43 asks the test participants if they generally had a positive attitude toward 
LeatherNet prior to using the system and question 44 asks them if their attitude is more 
positive after using the system. As might be expected, the user's attitude is generally neutral 
towards LeatherNet prior to using the system which indicates a lack of bias for or against the 
system prior to use. In fact, none of the respondents disagreed with the statement that their 
attitude was generally positive although nearly 50% expressed a neutral opinion. The 
respondent's attitude towards LeatherNet was much more positive overall after exposure to 
the system with 77% of the test participants indicating a more positive attitude. Only one 
participant stated that his attitude was less positive after exposure. The attitudes expressed 
here tend to indicate high user acceptance of LeatherNet as a mission planning and briefing 


tool. The distribution of the responses to questions 43 and 44 are shown in Charts 10 and 11 
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O43 
Chart 10. Assessment of the Respondents Attitude Towards LeatherNet Prior to 


Using the System 





Chart 11. Assessment of Respondent’s Attitude Towards LeatherNet After 


Exposure to the System 
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B. USER PERFORMANCE EVALUATION 





I. Performance Testing Results 

A total of 12 rifle companies from three regular infantry battalions and one reserve 
infantry battalion had their performance evaluated on Range 400 in connection with this 
evaluation of LeatherNet. One rifle company from each battalion served as the control group 
and received no LeatherNet training. Therefore, four control units and eight experimental 
units were sampled. Using the t test to test the null hypothesis, that no difference exists 
between the performance means of the two groups, revealed no statistically significant 
difference. The t value was calculated to be .179 with n - 1 degrees in each sample so that 
the total degrees of freedom equaled 10. Using at distribution table, this value was compared 
to the critical value that ensures a 95% confidence that we reject the null hypothesis only 
when it is indeed false. That critical value was 2.228. The t value must be greater than the 
critical value to reject the null hypothesis. Therefore, we must fail to reject the null 
hypothesis at this level of confidence. 

The control units averaged 78.5% of their objectives successfully completed while 
the experimental units achieved 79.6%. The small sample size can skew the results 
particularly in the case of the control units. Three of the control units averaged 71% of their 
objective achieved but the fourth control unit achieved 100%. The ability of this one unit to 
influence the overall results suggests the need for a larger sample size to ensure that the 
sample closely reflects the true characteristics of the population as a whole. If this one unit 
was eliminated from the study, the t test would reveal a statistically significant difference 
in the means at the 90% confidence level, but still not at the 95% level. Due to the fact that 
the performance evaluation relied upon the opinion of a single SME, the possibility also 
exists for the bias of the evaluator to have a profound effect on the test results as can be seen 
with the impact of one unit scoring 100%. While it is not being suggested that that is the 
case here, the possibility always exists in a subjective evaluation. 

Other reasons may exist for the lack of variance between the experimental and 
control units performance. As mentioned earlier, lack of experimental control permitted 


external influences upon both the experimental and control units. Access to additional 
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information concerning the Range 400 exercise by the control units, such as the ability to 
conduct a physical reconnaissance, would tend to reduce and even negate any information 
advantage that the experimental units received from LeatherNet. Additionally, having more 
and even better information doesn't mean that the information is put to its best use. While 
the user surveys indicate that a large portion of the test participants gained critical 
information, and that the information influenced their final attack plans, the participants still 
must possess the necessary skills to put that information to effective use. For instance, if a 
test participant learned from LeatherNet that certain avenues of approach were vulnerable 
to enemy fires, the participant still must find a more suitable alternative and execute that 
alternative with skill to realize a positivé effect. Major Newell, reflecting on the ability of 
commanders to use information presented to them during CAX training, stated; "giving a 
man tools doesn't make him a carpenter, but rather the skill with which he uses those tools." 
Having not been able to use LeatherNet to war game their plans, the test participants were 
left with only their normal conventional skills and imagination to finish planning and execute 
their plan of attack. Finally, the test participant were not free to exercise all the possible 
options in conducting the Range 400 exercise due to safety constraints and range regulations 


which limit their ability to take advantage of the information gained through LeatherNet. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. MISSION PLANNING AND BRIEFING CAPABILITIES 

1. User Acceptance 

The user surveys indicate that there is a high level of acceptance among the test 
participants for LeatherNet as a mission planning and briefing tool. The survey respondents 
generally agree that LeatherNet improves their ability to conduct an estimate of the situation 
and provides vital information. High user acceptance is supported by the large percentage 
of users who said the system provides critical information and that the information they 
received from LeatherNet influenced their final attack plan. The fact that 50% would risk 
altering their plan of attack based on information from a system they had no previous 
exposure to strongly suggests that the functionality of LeatherNet for mission planning 
purpose appears both authoritative and reliable. Further evidence of high user acceptance is 
derived from the fact that over 90% of all respondents agree that LeatherNet has the potential 
to be an effective mission planning tool, provides vital information in evaluating tactical 
alternatives, and demonstrates the potential to be an effective training tool for training 
company commanders and their staffs. Additionally, over 75% of the respondents had a 
more positive attitude toward the system after seeing it in operation. 

2. Benefits Derived from LeatherNet 

As mentioned, the users believe that the system provides critical information that was 
not known to them prior to using LeatherNet. The significance of this is that the respondents 
had ample opportunity to gain information concerning the Range 400 exercise from many 
other traditional sources, but still found that LeatherNet provided additional unknown 
information that was of a critical nature. The respondents indicated the ability to conduct 
"terrain walks" and the ability to display both enemy and friendly fire fans to be the most 
useful aspects of using LeatherNet to conduct an estimate of the situation. While they found 
LeatherNet useful for gaining information regarding most elements of an estimate of the 
situation they were less positive or expressed neutral sentiment that LeatherNet helped them 


understand obstacles, enemy avenues of approach for counterattacks, and ammunition 
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requirements. Finally, the test participants expressed nearly universal belief that LeatherNet 
has the potential to be an effective mission planning and COA analysis tool, and that it is 
effective for training unit leaders at the company level. 

Unfortunately, no performance benefits could be quantified from using LeatherNet 
according to the results from the Range 400 evaluations despite the positive user appraisal 
of the system. It was not intended that empirical testing be conducted with a system under 
development and without the ability to exercise the necessary experimental control. Due to 
the cost, resources, and time necessary to perform this type of testing, it 1s neither desired 
nor practical at this point. It was thought, however, that early user testing prior to full 
completion would provide the opportunity for user feedback while the system remains under 
development. When this research began it was expected that a more comprehensive test 
could have been performed. However, developmental issues, resistance to using CAX units 
for more comprehensive testing, and the limited availability of CAX units to conduct the 
tests all combined to limit the ability to fully test the system. Several lessons, however, have 


been gained that can improve the quality of the test process for LeatherNet in the future. 


B. RECOMMENDATIONS 

1. System Testing 

The Marine Corps is continuing to increase its interest and investment in the use of 
simulation technology to better prepare and train its forces. LeatherNet represents only a 
small portion of the overall effort and is primarily designed to be a demonstration project 
with a secondary purpose of providing an intuitively appealing training tool for CAX units. 
If the initial feedback from the survey conducted in conjunction with this research is any 
indication, LeatherNet has a good potential to successfully fulfill a role as a valuable training 
tool. How well LeatherNet may demonstrate its capabilities and technologies remains to be 
seen. While LeatherNet observers may have a qualitative feel that the system can provide 
effective mission planning, mission rehearsal, and "what if" COA analysis, there seems to 
be little or no data to quantify LeatherNet's effectiveness. Admittedly, it has proven to be a 


difficult task to quantify the training effectiveness of a virtual simulation system, particularly 


58 





one designed for collective training. In fact it is hard to quantify the training effectiveness 
of collective training alone, never mind evaluating it in combination with simulation. 

In order to quantify the training effectiveness of virtual simulators such as 
LeatherNet and TTES, it is recommended that the Marine Corps follow the Army’s lead in 
conducting user testing.. The Army is aware of the deficiency in quantifying the training 
effectiveness of its training simulators and has committed itself to conducting the most 
comprehensive test of a simulation system to date with its CCTT system. Although 
LeatherNet may be primarily a demonstration project, the best way to demonstrate the 
project is to quantify the results. The ability to say the system can save units X dollars while 
improving their ability to successfully complete Y training objectives by Z percent is a much 
more powerful promotion tool than simply saying we believe it improves training, based on 
user feedback. While it is recognized that the resources for testing most likely will only 
allow for a much scaled down version of the Army’s test plans, tests can still be designed 
to provide quantitative data. Such data can be used to provide solid justification regarding 
future M&S investments. 

For follow on tests, it is recommended that dedicated test units be assigned to the 
project during the test period, rather than using units distracted by other training and 
command commitments. Undedicated test units will tend to give the test process lower 
priority than their operational demands, and the ability to conduct the test properly will 
suffer accordingly. Secondly, experimental control needs to be exercised over the test units 
to eliminate outside sources of influence that can corrupt the results. Neither the control nor 
the experimental units should have access to outside information and assistance other than 
that information and assistance which the evaluators desire to allow each unit. Particularly, 
in the case of an exercise such as Range 400, neither group should be allowed to do a 
reconnaissance of the enemy held terrain nor collect advice from others who have just 
completed the exercise. Denying the control units the ability to collect intelligence not 
normally available on a battlefield will more likely show how much more valuable a virtual 
simulator can be in preparing for an exercise compared to preparing without if it 1s in fact 


more valuable. Conducting a true experiment, consisting of a pre-test to determine skill 


59 





levels, training on LeatherNet, and a post-test to determine post-training skill levels for the 
experimental group and a pre-test and post-test for the control group should be considered. 
Finally, incorporation of instrumented ranges, such as those being developed for the Emerald 
Light Project to assess the performance of the units in both the pre-test and post-test phases 
should also be considered. Instrumented ranges can provide data analysis of position and 
tracking of indirect fires and position and trigger-pulls of ground entities and direct fires. 
Developing a methodology to use this analysis to rate performance provides the possibility 
to eliminate, to a large degree, the subjective nature of evaluating collective training. 

While this type of an experiment will most likely be expensive and time consuming 
to conduct, the units will be able to gain worthwhile training while providing valuable test 
data. Such data can also assist the Marine Corps modeling and simulation effort by providing 
information on how to improve our training simulators and where to focus our M&S 
development and procurement dollars . 

Z3 Expansion of M&S Resources 

The use of virtual simulation systems holds great promise to provide unique 
opportunities to overcome the current and future obstacles to training such as high cost and 
loss of training areas. The great majority of the test participants agreed that it would have 
been beneficial to have been able to use the LeatherNet system at their home base, to have 
been allowed more extensive use of the system, and to include more of their personnel in the 
simulation training. To make these desires a reality, the Marine Corps should continue to the 
development and implementation of both demonstration and training sites of DIS compliant 
simulators. Doing so will enable Fleet Marine Force (FMF) commands to train with and be 
exposed to what promises to be an increasingly important training alternative for combat 
forces. Such exposure can promote acceptance among the ranks of both our junior and senior 
leadership and perhaps smooth the way for further implementation. 

Expansion of the terrain databases to incorporate other areas, particularly littoral 

areas is essential to enhancing the training value of virtual simulators such as LeatherNet for 
the Marine Corps. Current efforts to develop the Littoral Warfare Training Complex, 


including instrumented ranges, data processing and display capabilities, and the required 
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communication infrastructure is a step in the right direction. Without the modeling of the 
littoral terrain as well as the associated Marine Corps equipment, tactics, and command and 
control characteristics, the Marine Corps risks being left out as an effective participant in 
joint simulation initiatives and exercises. The only way to remain relevant in these initiatives 
is to devote the necessary resources to develop those systems and functionality that cannot 
be leveraged from the other services. | 

The Marine Corps faces a variety of training constraints that could potentially 
degrade the readiness of its forces. Investing in DIS compliant simulation systems will 
ensure that the FMF commands have access to world class training opportunities while 
preserving resources, in the long run, to conduct mission essential live fire training exercises 
such as those associated with the CAX program. The longer the Marine Corps takes to 
investigate the various simulation options and invest where the opportunities appear to be 
the most promising, the longer those systems will be denied to those forces that might 


benefit from them. 
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APPENDIX A. LEATHERNET USER SURVEY 
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7 April 96 


Fellow Marine, 


Your unit has been randomly selected to participate in user testing of LeatherNet. An 
important part of the process is a survey of your opinion toward this simulator and its value 
as a mission planning and mission training device. The survey is part of a study being 
conducted at the Naval Postgraduate School to determine the value of virtual reality 
simulators in helping ground units to train and prepare for combat missions. 


Currently, LeatherNet represents the only operational simulator of its kind in the U.S. 
Marine Corps. Therefore, the results of this survey are important because they are the only 
data available concerning what the "Grunts" think of the system. 


All responses will be kept confidential. Only the survey data will be compiled and no 
personal or unit data will be released from this survey. 


Please answer all questions to the best of your ability. Thank you for your cooperation. 


Tracy R. Hague 
Major USMC 
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RANGE 400 / LEATHERNET LAB QUESTIONNAIRE 


CAX #: 
DATE IN LEATHERNET LAB: 


DATE ON RANGE 400: 


I. Experience Questions 


1. How many years of active military service do you have? 


O Fewer than 3 years 
O 3 to 6 years 

O 6to 9 years 

O 9 to 12 years 

O More than 12 years 


2. How long have you held your current billet? 
© Fewer than 6 months 
O 6to 12 months 
O 12 to 18 months 
© 18 to 24 months 
O Longer than 24 months 
3. Have you personally run range 400 anytime prior to this CAX”? 


O Yes 
O No 


If yes, when did you last run range 400? 


4. How often does your company conduct live fire and maneuver exercises? 
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O Fewer than once a year 

O 1 to 3 times a year 

O 4 to 6 times a year 

© 7 to 10 times a year 

O More than 10 times a year 
O Don't know 


5. How would you rate your company's overall tactical proficiency? 


© Far below average 
O Below average 

O Average 

O Above average 

O Far above average 


6. Did you physically walk any of the Range 400 terrain forward of Machine Gun 
Hill prior to the live fire exercise? 


© Yes 
O No 


7. Did you receive advice on how to conduct the Range 400 exercise from other 
Marines in your unit who had previous experience on Range 400? 


© Yes 
O No 


8. Did you observe other units conduct the Range 400 exercise during the current 
CAX prior to your company's Range 400 exercise? 


O Yes 
O No 
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II. LeatherNet User Questions 
A. Ability to conduct an estimate of the situation 


Questions 8 through 23 refer to the statement below. 


The LeatherNet lab improved my understanding of the following information: 
Enemy situation 


9. Enemy positions 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


10. Enemy fields of fire 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 
Terrain 


11. Friendly avenues of approach 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 
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The LeatherNet lab improved my understanding of the following information: 


12. Obstacles 


Strongly Disagree Neither agree 
disagree nor disagree 
O O O 


13. Enemy avenues of approach for counterattacks 


Strongly Disagree Neither agree 
disagree nor disagree 


O O O 


14. Potential friendly firing positions 


Strongly Disagree Neither agree 
disagree nor disagree 
Oo » O O 
Fire support 


15. Fields of fire for friendly weapons from potential firing positions 


Strongly Disagree Neither agree 
disagree nor disagree 
O O O 


Agree 


O 


Agree 


Agree 


Agree 


O 


16. Range fans for friendly weapons from potential firing positions 


Strongly Disagree Neither agree 
Disagree nor disagree 


O O O 
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Agree 


O 


Strongly 
agree 


O 


Strongly 
agree 


O 


Strongly 
agree 


O 


Strongly 
agree 


O 


Strongly 
agree 


O 





OO 


The LeatherNet lab improved my understanding of the following information: 


17. Mortar sheaf's ability to suppress enemy positions 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


18. Machine gun target coverage 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


19. Shift/cease fire points 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O oO O O O 


20. Weapon safety fans 


Strongly Disagree Neither agree | Agree Strongly 
disagree nor disagree agree 
O O O O O 
Time/Space 


21. Time exposed to enemy fire along avenues of approach 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 
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The LeatherNet lab improved my understanding of the following information: 


22. Time required to execute movements along avenues of approach 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 
Logistics 


23. Ammunition requirements 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


B. Overall effectiveness of LeatherNet 


24. The simulated terrain in LeatherNet closely matched the Range 400 terrain. 


Strongly Disagree Neither agree | Agree Strongly 
disagree nor disagree agree 
O O O O O 


25. The LeatherNet lab influenced my final attack plan. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


26. The LeatherNet lab provided critical information that was previously 
unknown to me. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 
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27. LeatherNet has the potential to be an effective mission planning tool. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


28. LeatherNet has the potential to be an effective mission rehearsal tool. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


29. LeatherNet provides vital information in evaluating tactical alternatives. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


30. LeatherNet demonstrates the potential to be an effective tool for training company 
commanders and their staffs. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 
31. The LeatherNet lab would be more useful if we could use it at our home base 
just prior to CAX . 
Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


32. The LeatherNet lab would be more useful if we had more time to use the 
system to evaluate our plans. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 
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33. 


34. 


35. 


36. 


37. 


The LeatherNet lab would be more useful if more personnel in my unit had the 
opportunity to use the system. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


The LeatherNet lab would have been of more value in preparing for Range 400 if 
we did not have other means of obtaining information concerning the exercise such 
as Our Own reconnaissance or our fellow Marine's advice. 


Strongly _— Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


What other changes can be made to make LeatherNet more useful? 




















What were the most useful features of LeatherNet? 




















What were the least useful features? 
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38. Please rank the following capabilities in order of importance ( 1 to indicate the 
most important, 8 to indicate the least important) Please note that you may 


have not been exposed to all of these capabilities. If that is the case, your ranking 
should reflect how important you believe it would be to have these capabilities. 


A. Ability to war game against a realistic, responsive enemy 


B. Quantitative feedback on effectiveness of weapons employment 


C. Ability to reconstruct/replay an exercise for detailed debrief 
D. Ability to conduct "danger close" fire support training 
E. Ability to evaluate a subordinate's performance 


F. Ability to train on simulated terrain in the absence of the real 
terrain (ie: aboard a ship) 


G. Ability to conduct the same exercise under varying conditions 
or to modify a plan and evaluate its outcome 


H. Simulated casualty assessment 


a ed 


39. It would have been beneficial to evaluate our plan of attack using all of the features 


listed in question 37. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


40. What other features would you like to see developed for LeatherNet? 
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42. 


43. 


44. 





I felt uncomfortable relying on LeatherNet to evaluate my plan of attack 
because I was unfamiliar with computer simulators. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


I did not change my plan of attack because I was not sure I could trust the 
results of the LeatherNet lab in a real exercise. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


My attitude toward LeatherNet was generally positive prior to conducting the 
LeatherNet lab. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


My attitude toward LeatherNet is more positive now that I have seen it in 
operation. 


Strongly Disagree Neither agree Agree Strongly 
disagree nor disagree agree 
O O O O O 


Please return this survey to Major Newell or the survey coordinator. 
Thank you again for your cooperatio 
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APPENDIX B. RANGE 400 EVALUATION FORMS 


fs 


UNIT DATE 
BASES OF FIRE/FIRING POSITION SELECTION. 


Appropriate ranges to enemy positions. 


Heavy machine guns YES NO 

Machinegun section YES 
NO 

Dragons. YES NO 

Rifle platoon bases of fire. _ YES NO 
COMMENTS: 


ROUTE SELECTION. 


Covered/concealed routes used to occupy Base of Fire/firing positions? 


Heavy machine guns. YES NO 

Machinegun section. YES NO 

Dragons. YES NO 

Rifle platoon bases of fire. YES NO 
COMMENTS: 
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Covered and concealed avenue of approach used for assault element. 
YES NO 


COMMENTS: 














SUPPORT OF MANEUVER. 


Suppressive fires used in support of occupation of Base of Fire/firing positions. 


Machinegun section. YES NO 

Dragons. YES NO 

Rifle platoon bases of fire. YES NO 
COMMENTS: 














Suppressive fire delivered on all enemy positions that could engage exposed 
movement of the assault element. 


Breach of minefield and movement into wash. YES NO 
Breach of wire. YES NO 
Assault of western position. | YES NO 
Assault of center position. YES NO 
Assault of CP position. YES NO 


TT 











COMMENTS: 














Obscuration delivered on enemy positions that could deliver fire on obstacle breaches. 


Breach of minefield. YES NO 
Breach of wire. YES NO 
COMMENTS: 














Counterfire delivered on enemy mortar position. 


At beginning of attack/pre-H-Hour. YES NO 
During breach of wire obstacle. YES NO 
COMMENTS: 














Delivery of suppressive fires effective. 
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Adequate coverage of enemy positions (i.e. mortar sheafs). YES NO 
Appropriate machinegun gunnery techniques employed. YES NO 
COMMENTS: 


Fires shifted/ceased at appropriate times (by the unit, as close as possible without 
violating safety fans). 


YES NO 


COMMENTS: 


No unnecessary fires (fires on enemy positions unable to engage friendlies). 
YES NO 


COMMENTS: 


Firing units did not run out of ammunition before their mission was complete. 


79 


60mm mortars YES NO 

81mm mortars YES NO 

Machinegun section YES NO 

Heavy machine guns YES NO 

Assault platoons YES NO 
COMMENTS: 


a 
OO 
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Appropriate positions were selected to observe/adjust mortar fires (STA/FST). 


YES NO 


COMMENTS: 

a 
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